Automated Audio Captioning (AAC) aims to describe the semantic contexts of general sounds, including acoustic events and scenes, by leveraging effective acoustic features. To enhance performance, an AAC method, EnCLAP, employed discrete tokens from EnCodec as an effective input for fine-tuning a language model BART. However, EnCodec is designed to reconstruct waveforms rather than capture the semantic contexts of general sounds, which AAC should describe. To address this issue, we propose CLAP-ART, an AAC method that utilizes "semantic-rich and discrete" tokens as input. CLAP-ART computes semantic-rich discrete tokens from pre-trained audio representations through vector quantization. We experimentally confirmed that CLAP-ART outperforms baseline EnCLAP on two AAC benchmarks, indicating that semantic-rich discrete tokens derived from semantically rich AR are beneficial for AAC. Index T erms: automated audio captioning, discrete token, residual vector quantization, audio representation.

In this technical report, we describe our submission to DCASE2024 Challenge Task6 (Automated Audio Captioning) and Task8 (Language-based Audio Retrieval). We develop our approach building upon the EnCLAP audio captioning framework and optimizing it for Task6 of the challenge. Notably, we outline the changes in the underlying components and the incorporation of the reranking process. Additionally, we submit a supplementary retriever model, a byproduct of our modified framework, to Task8. Our proposed systems achieve FENSE score of 0.542 on Task6 and mAP@10 score of 0.386 on Task8, significantly outperforming the baseline models.

While great effort has been done, the data scarcity issue In recent years, datasets of paired audio and captions have enabled of audio captioning still withholds. The common datasets in AAC, remarkable success in automatically generating descriptions AudioCaps and Clotho, contain together 50k captions for training, for audio clips, namely Automated Audio Captioning (AAC). However, whereas 400k captions are provided in COCO caption [8] for image it is labor-intensive and time-consuming to collect a sufficient captioning. Kim et al. [9] observe that due to the limited data, prior number of paired audio and captions. Motivated by the recent arts design decoders with shallow layers that fail to learn generalized advances in Contrastive Language-Audio Pretraining (CLAP), language expressivity and are fitted to the small-scaled target we propose a weakly-supervised approach to train an AAC model dataset. Due to this issue, their performance radically decreases assuming only text data and a pre-trained CLAP model, alleviating when tested on out-of-domain data. Motivated by these limitations the need for paired target data. Our approach leverages the we present an approach to AAC that only requires a pre-trained similarity between audio and text embeddings in CLAP.

This project involved participation in the DCASE 2022 Competition (Task 6) which had two subtasks: (1) Automated Audio Captioning and (2) Language-Based Audio Retrieval. The first subtask involved the generation of a textual description for audio samples, while the goal of the second was to find audio samples within a fixed dataset that match a given description. For both subtasks, the Clotho dataset was used. The models were evaluated on BLEU1, BLEU2, BLEU3, ROUGEL, METEOR, CIDEr, SPICE, and SPIDEr scores for audio captioning and R1, R5, R10 and mARP10 scores for audio retrieval. We have conducted a handful of experiments that modify the baseline models for these tasks. Our final architecture for Automated Audio Captioning is slightly better than the baseline performance, while our model for Language-Based Audio Retrieval has surpassed its counterpart.

This technical report describes the system participating to the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 Challenge, Task 6: automated audio captioning. Our submission focuses on solving two indeterminacy problems in automated audio captioning: word selection indeterminacy and sentence length indeterminacy. We simultaneously solve the main caption generation and sub indeterminacy problems by estimating keywords and sentence length through multi-task learning. We tested a simplified model of our submission using the development-testing dataset. Our model achieved 20.7 SPIDEr score where that of the baseline system was 5.4.